52 research outputs found
Fixed-Form Variational Posterior Approximation through Stochastic Linear Regression
We propose a general algorithm for approximating nonstandard Bayesian
posterior distributions. The algorithm minimizes the Kullback-Leibler
divergence of an approximating distribution to the intractable posterior
distribution. Our method can be used to approximate any posterior distribution,
provided that it is given in closed form up to the proportionality constant.
The approximation can be any distribution in the exponential family or any
mixture of such distributions, which means that it can be made arbitrarily
precise. Several examples illustrate the speed and accuracy of our
approximation method in practice
The Likelihood of Mixed Hitting Times
We present a method for computing the likelihood of a mixed hitting-time
model that specifies durations as the first time a latent L\'evy process
crosses a heterogeneous threshold. This likelihood is not generally known in
closed form, but its Laplace transform is. Our approach to its computation
relies on numerical methods for inverting Laplace transforms that exploit
special properties of the first passage times of L\'evy processes. We use our
method to implement a maximum likelihood estimator of the mixed hitting-time
model in MATLAB. We illustrate the application of this estimator with an
analysis of Kennan's (1985) strike data.Comment: 35 page
Variable Selection and Functional Form Uncertainty in Cross-Country Growth Regressions
Regression analyses of cross-country economic growth data are complicated by two main forms of model uncertainty: the uncertainty in selecting explanatory variables and the uncertainty in specifying the functional form of the regression function. Most discussions in the literature address these problems independently, yet a joint treatment is essential. We perform this joint treatment by extending the linear model to allow for multiple-regime parameter heterogeneity of the type suggested by new growth theory, while addressing the variable selection problem by means of Bayesian model averaging. Controlling for variable selection uncertainty, we confirm the evidence in favor of new growth theory presented in several earlier studies. However, controlling for functional form uncertainty, we find that the effects of many of the explanatory variables identified in the literature are not robust across countries and variable selections
Variational Dropout and the Local Reparameterization Trick
We investigate a local reparameterizaton technique for greatly reducing the
variance of stochastic gradients for variational Bayesian inference (SGVB) of a
posterior over model parameters, while retaining parallelizability. This local
reparameterization translates uncertainty about global parameters into local
noise that is independent across datapoints in the minibatch. Such
parameterizations can be trivially parallelized and have variance that is
inversely proportional to the minibatch size, generally leading to much faster
convergence. Additionally, we explore a connection with dropout: Gaussian
dropout objectives correspond to SGVB with local reparameterization, a
scale-invariant prior and proportionally fixed posterior variance. Our method
allows inference of more flexibly parameterized posteriors; specifically, we
propose variational dropout, a generalization of Gaussian dropout where the
dropout rates are learned, often leading to better models. The method is
demonstrated through several experiments
Blurring Diffusion Models
Recently, Rissanen et al., (2022) have presented a new type of diffusion
process for generative modeling based on heat dissipation, or blurring, as an
alternative to isotropic Gaussian diffusion. Here, we show that blurring can
equivalently be defined through a Gaussian diffusion process with non-isotropic
noise. In making this connection, we bridge the gap between inverse heat
dissipation and denoising diffusion, and we shed light on the inductive bias
that results from this modeling choice. Finally, we propose a generalized class
of diffusion models that offers the best of both standard Gaussian denoising
diffusion and inverse heat dissipation, which we call Blurring Diffusion
Models
- …